Skip to content

[Real-time Table Replication X clusters][2/n] Create a Controller Job for historical segments backfill#17521

Open
wirybeaver wants to merge 16 commits intoapache:masterfrom
wirybeaver:oss/tableCopyBackfill
Open

[Real-time Table Replication X clusters][2/n] Create a Controller Job for historical segments backfill#17521
wirybeaver wants to merge 16 commits intoapache:masterfrom
wirybeaver:oss/tableCopyBackfill

Conversation

@wirybeaver
Copy link
Contributor

@wirybeaver wirybeaver commented Jan 17, 2026 •
edited
Loading

  • Create an async controller zk jobs to deeply backfill historical segments prior to the inducted consumer watermarks. The deep copy means replicating segments to a standalone remote storage path.
  • Introduced TableReplicator to fetch segment ZK metadata from the source, register a controller job, and submit 4 worker tasks (can be fine tuned in the future) that call a SegmentCopier for each segment; errors immediately trigger job updates and completed segments are batched for ZK updates.
  • Added ZkBasedTableReplicationObserver and TableReplicationProgressStats to keep in-memory progress (remaining count and failed segment list) and persist job metadata to ZK (including a SEGMENTS_TO_BE_COPIED field) every 100 completions or immediately on error.
  • Implemented SegmentCopier interface with RealtimeSegmentCopier (deep-copy mode) that copies segments between the same PinotFS scheme and uploads the segment URI to the destination controller; if a segment lacks a deep-store download URL the code logs the problem and records failures.

@wirybeaver wirybeaver changed the title Oss/table copy backfill [Real-time Table Replication X clusters][2/n] Create a Controller Job for historical segments backfill Jan 17, 2026
@wirybeaver wirybeaver mentioned this pull request Jan 17, 2026
Open
2 tasks
@wirybeaver wirybeaver force-pushed the oss/tableCopyBackfill branch 2 times, most recently from 79ff96a to 91b71db Compare January 22, 2026 05:04
@codecov-commenter
Copy link

codecov-commenter commented Jan 22, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 65.61086% with 76 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.21%. Comparing base (9de73ec) to head (5167f4f).
⚠️ Report is 12 commits behind head on master.

Files with missing lines Patch % Lines
.../helix/core/replication/RealtimeSegmentCopier.java 57.57% 23 Missing and 5 partials ⚠️
...roller/helix/core/replication/TableReplicator.java 60.41% 17 Missing and 2 partials ⚠️
...ntroller/helix/core/PinotHelixResourceManager.java 42.30% 13 Missing and 2 partials ⚠️
...oller/api/resources/PinotTableRestletResource.java 0.00% 9 Missing ⚠️
...e/replication/ZkBasedTableReplicationObserver.java 90.62% 2 Missing and 1 partial ⚠️
...ller/helix/core/replication/NoOpSegmentCopier.java 0.00% 2 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##             master   #17521      +/-   ##
============================================
- Coverage     63.24%   63.21%   -0.03%     
  Complexity     1454     1454              
============================================
  Files          3176     3182       +6     
  Lines        191002   191219     +217     
  Branches      29204    29223      +19     
============================================
+ Hits         120791   120888      +97     
- Misses        60796    60913     +117     
- Partials       9415     9418       +3
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-11 63.17% <65.61%> (-0.03%) ⬇️
java-21 63.19% <65.61%> (+0.02%) ⬆️
temurin 63.21% <65.61%> (-0.03%) ⬇️
unittests 63.21% <65.61%> (-0.03%) ⬇️
unittests1 55.58% <0.00%> (+0.01%) ⬆️
unittests2 34.11% <65.61%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@wirybeaver wirybeaver force-pushed the oss/tableCopyBackfill branch 3 times, most recently from cb4267f to b4b9ec1 Compare January 26, 2026 06:21
@wirybeaver wirybeaver force-pushed the oss/tableCopyBackfill branch from b4b9ec1 to 9c2eb01 Compare February 2, 2026 04:12
@wirybeaver
add http:// to the uri
4084ebc
@wirybeaver
enforce user input http:// or https://
8bbe56f
@wirybeaver
[Pinot Copy Table] Create a Controller Job for historical segments ba…
b536ae6 
…ckfilling
@wirybeaver
persist watermarks in zk; record number of segments to copy instead o…
235e4cd 
…f the actual list; add job status in zk
@wirybeaver
make WatermarkInductionResult fields private
340d468
@wirybeaver
segment v2 push request with retry policy
673974f
@wirybeaver
add endpoint to fetch job status
997de4e
@wirybeaver
add tableName as a path parameter
3e0ecc2
@wirybeaver
forget to add v2/segments path
15ea8ec
@wirybeaver
need to specify the table type for REALTIME
382d8cc
@wirybeaver
fix the compilation error due to long -> int change in previous PR
65c86f4
@wirybeaver
the claude code fails to pick up the important integration of table r…
302eec4 
…eplicator
@wirybeaver
make parallism tunable
741c6ae
@wirybeaver
get uri for zk metadata
dcd05eb
@wirybeaver
fix the unit test failure
7a9e104
@wirybeaver wirybeaver force-pushed the oss/tableCopyBackfill branch from 9c2eb01 to 7a9e104 Compare February 22, 2026 00:39
@wirybeaver wirybeaver force-pushed the oss/tableCopyBackfill branch from 8e4f21a to 5167f4f Compare February 24, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

feature ingestion real-time

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wirybeaver @codecov-commenter @Jackie-Jiang